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THE INTERNAL MODEL: A STUDY OF THE RELATIVE CONTRIBUTION 
OF PROPRIOCEPTION AND VISUAL INFORMATION TO FAILURE 
DETECTION IN DYNAMIC SYSTEMS* 

By Colin Kessel and Christopher D. Wickens 
Department of Psychology, University of Illinois 

SUMMARY 


The development of the internal model as it pertains to the detection of 
step changes in the order of control dynamics Is investigated for two modes 
of participation: whether the subjects are actively controlling those dy- 

namics or are monitoring an autopilot controlling them, A transfer of train- 
ing design was used to evaluate the relative contribution of proprioception 
and visual information to the overall accuracy of the Internal model. Six- 
teen subjects either tracked or monitored the system dynamics as a 2-dimen- 
sional pursuit display under single task conditions and concurrently with a 
"sub-critical" tracking task at two difficulty levels. Detection performance 
was faster and auire accurate in the manual as opposed to the autopilot mode. 
The concurrent tracking task produced a decrement in detection performance for 
all conditions though this was more marked for the manual mode. The develop- 
ment of an internal model in the manual mode transferred positively to the 
automatic mode producing enhanced detection performance. There was no trans- 
fer from the internal model developed in the automatic mode to the manual 
mode. 


INTRODUCTION 


Over the past few years there has been a great deal of research directed 
at the problem of determining the differences between operators and monitors 
of dynamic systems (References 1-7), While the conclusions reached by these 
authors do not always coincide, there is a general consensur that a greater 
understanding of the different processes operating in the two modes of parti- 
cipation is necessary for the successful integration of automated systems in 
the workplace. 

We have provided a detailed theoretical analysis of the processes in- 
volved in the two modes of participation (Reference 7), Briefly, this anal* 
ysis has argued that one way in which the differences between modes of parti- 
cipation can be studied is by determining the relative sensitivity of operators 
versus monitors in a failure detection task. 

"this research was funded by the Life Sciences Program, Air Force Office 
of Scientific Research, Contract Number F44620- 76-C-0009 . Dr. Alfred 
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Three attribute, were Identified that would seemingly facilitate failure 
detection in the controlling mode: (i) a smaller variability of the Internal 

<odel of the system; (ii) the options of testing hypotheses about the nature 
of the dynamics by introducing signals into the system; and (ill) a greater 
number of information channels available upon which to base failure detection 
decisions. It was recognised however, that this latter advantage may b« wit- 

f*« 4 d th * tlut: •) *d*ptation takes place reducing the strength 

of visual error information and, b) proprioceptive sensitivity is less than 


In comparison iTlie monitoring mode was also characterised by two attrl- 
butes that could facilitate detections: a greater "strength" of the visual 
signal (if adaptation by the autopilot does not take place) and a lower 
level of operator workload. 


* Bt “ dy conducted (Reference 7) to test the above theoretical analysis 

found that detection performance in the manual mode was faster and only 
slightly less accurate than the autopilot mode. Furthermore the observed 
manual superiority was attributed to the additional proprioceptive informa- 
tion resulting from operator control adaptation to the system change. It 
s possible that some contribution to manual mode superiority in our prior 

H^fver e rKi. e h 8refct<r »°del consistency in that mode. 

However this hypothesis was .sensed to be doubtful because a within subjects 

and ^manual *° Si** ***** * ubJe< ** Participated in both automatic 

and manual conditions. Thus the internal mocel developed in manual condi- 

"2J 7*™ pr * W “ #bly b * aval *« b I* to facilitate detection in the automatic 


• getter distinction between the internal model 
employed in the two modes, the present study employed a between subject de- 
sign using a transfer of training technique. This procedure enables an 
examination of the development of internal models, in the two modes of par- 
ticipation, and subsequently measures their impact upon transfer tc the other 


ntrfJLn!! ? yP S J d t h this technique would increase the differential 
performance in detection between the two modes of participation while at the 
same time demonstrating that the internal model developed in the manual mode 
^r£ownc^*' ** * utiU ** d to Militate automatic mode failure detection 


METHOD 


Subjects : 

The subjects were 18 right-handed male university students. Subjects 
were paid a base rate of $2 .SO per hour but could increase tneir average 
pay by maintaining a high level of detection performance. 
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Apparatus : 

The basic experimental equipment included a 7.5 x 10 cm Hewlett Packard 
Model 1300 CRT display, a spring* centered, dual-axis tracking hand control 
with an index-finger trigger operated with the right hand, and a spring 
loaded finger controller operated with the left. A Raytheon 704 16-bit dig- 
ital computer with 24k memory and A/D, D/A interfacing was used both to gen- 
erate inputs to the tracking display and to process responses of the subject. 
The subject was seated on a chair with two arm rests, one for the tracking 
hand controller and one for the side-task finger controller. The subject's 
eyes were approximately 112 centimeters from the CRT display so that the dis- 
play subtended a visual angle of 1.5°. 

Tracking tasks. The primary pursuit-tracking task required the subject 
to match the position of a cursor with that of a target which followed a 
semi-predictable two-dimensional path across the display. The target's 
path was determined by ihe summation of two non-harmonica lly related sinus- 
oids (.05 and .08 Hz) along each axis with a phase offset between the axes. 
The position of the following cursor was controlled Jointly by the subject's 
control response and by a band-limited forcing function with a cutoff fre- 
quency of ,32 Hz for both axes. Thus the two inputs to the system were well 
differentiated in terms of predictability, bandwidth, and locus of effect 
(target vs. cursor). The control dynamics of the tracking task were of the 
form Y * 1 “ a + <* for each axis, where a was the variable 

parameter used to introduce changes in the system dynamics. These changes, 
or simulated failures, were introduced by step changes in the acceleration 
constant a from a normal value of ,3, a mixed velocity and acceleration sys- 
f*®# to a ■ ,9, a system that approximates pure second order dynamics that 
requires the operator to generate considerable lead in order to maintain 
stable performance. 


As the loading task, the Critical Task (Reference 8) was employed. 

This was displayed horizontally in the center of the screen and required the 
subject to apply force to the finger control in a left-right direction to 
maintain the unstable error cursor centered on the display. The value of 
the instability constant X in the dynamics _ k \ was set at a 


Two values (X 1 


c s - x 

0,5 and X“ 1.0) were employed on 


constant subcrltlcal value, 
different dual task trials. 

Experimental Design and Task : 

Three groups were used in the transfer of training design (see Figure 1) 
Group one transferred from manual (MA ) on session one to automatic (AU ) on' 
session two; group two transferred from automatic (AU.) to manual (MA $*whili 
group three was the control group for the automatic condition and monitored 
in the automatic mode in both sessions (AU_ and AU..,..), The control 
group for the manual group (MA ) was MA..'Tne vario5l ( gtoup comparisons 
are represented in Figure 1 byfirrovs and will be referred to at greater 
length in the results section. 
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Each group participated In six 
consecutive days of data collection. 

These were divided into two sessions; 

3 days in each session with each ses- 
sion comprising 1 training day and 
two experimental days . Subjects in CR0UP , 
group ode for example participated in dual task i 
3 manual (MA_) sessions and then trans- 
ferred to 3 automatic (AU^) sessions. 


In the manual (MA) condition the 
subject performed the tracking manually 
while in the autopilot (AU) condition, 
his role in the control loop was re- 
placed by simulated autopilot control 
dynamics consisting of pure gain, ef- 
fective time delay, and a small added 
remnant. Each trial, MA or AU, lasted 
ISO seconds. 


CROUP 2 
OUAL TASK I 


GROUP 3 
OUAL TASK I 


Figure 1: 


SESSION £ SESSION H 



Experimental design and 
group comparisons 


Training Par: The training day 

was designed to give the subject maximum experience and practice with the 
system. Subjects therefore received extensive practice tracking (or moni- 
toring) with both prefailure and postfailure dynamics. Following this, 
they observed and then detected the step changes in dynamics. Practice with 
the critical side task was also included. 


The presentation of the failure was generated by an algorithm that as- 
sured random intervals between presentations and allowed the subject suffi- 
cient time to establish baseline tracking performance before the onset of 
the next change. Task logic also ensured that changes would only be intro- 
duced when system error was below a criterion value. In the absence of this 
precaution, changes would sometimes introduce obvious ''jumps" in cursor 
position. 

During the detection trials, the detection decision was recorded by 
pressing the trigger on the control stick* This response presented a "T" on 
the screen and returned the system to normal operating conditions of the pre- 
failure dynamics. If the subject failed to detect the change, the system re- 
turned to normal after six seconds via a 4 second ramp. On the basis of pre- 
test data, it was assumed that six seconds was the interval within which 
overt responses would correspond to detected failures and not false alarms. 
The subjects were told to detect as many changes as possible as quickly as 
possible . 

Experimental Days : The training day was followed by two consecutive 

experimental days. After four refresher trials in the AU or MA modes (de- 
pending upon the condition) with the side task, and a number of demonstrated 
failures, the subjects performed IS experimental tri*ls: 5 single task, 

tracking (or monitoring) only; 5 tracking with the easy critical task 
(X- o.5); and 5 tracking with the difficult critical task (V- 1.0). The 
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order of presentation was randomized. Each trial contained an average of 5 
failures per trial with a range of 4 to 6. 

The subject was instructed to "do the side task as efficiently and accur- 
ately as possible," and told to maintain that task at a standard level of per- 
formance. After each trial the subject received feedback about both his side 
task and detection performance. The instructions, feedback and payoff sche- 
dule, therefore, clearly defined the side task as the loading task while al- 
lowing the tracking and detection tasks to fluctuate in response to covert 
changes in available attentional resources (Reference 9). 

ANALYSIS 


Detection performance was assessed in terms of the accuracy and latency 
of responses. In computing the accuracy measure, signal detection theory 
analysis based upon the method of free response was employed (Reference 10). 
This technique accounts for the presence of hits and false alarms in the 
data; and the semi random occurance of failures within a trial. The area 
under the ROC curve (A[R0C]) was employed as the final accuracy measure (Ref- 
erence 11). Further details of this analysis procedure may be found in 
Wickens and Kessel (Reference 7, 12). 


The A(ROC) measure and the latency measure were then plotted in the form 
of a joint speed-accuracy measure depicted in Figure 2. "Good" performance 
is represented by points lying on the upper left, in the region of fast ac- 
curate response. Performance was quantified by projecting the point locus 
obtained onto the positive diagonal performance axis. The performance scale 
ia computed as (10 times A(R0C] • LATENCY) and will be called the "derived 
performance score." This procedure pro- 
duces a performance index that ranges 
from 0 for chance level of accuracy with 


a latency of 5" to 10.0 for perfect de- 
tection with 0 second reaction time. 

The units assigned to this performance 
index are somewhat arbitrary but are 
based on the observation that the over- 
all variability (standard deviation) of 
the raw latency scores were found to be 
roughly 10 times the variability of the 
A(R0C) measure* 

RESULTS 


Averages and standard deviations 
were computed for the accuracy (A[R0C]), 
the latency and the derived performance 
measures following the rational and the 
procedures outlined in the preceding 
section. 



Figure 2: Speed-accuracy repre- 

sentation of detection 
performance 
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The group averages for all chree measures are presented graphically in 
Figures 3 and 4* Figure 3 represents the results for the single task con* 
dition while Figure 4 represents the dual task, workload condition collapsed 
over both levels of dual task difficulty (the rationale for this procedure 
is discussed below). The symbols in Figures 3 and 4 represent the group re* 
suits in the speed*accuracy space, while the arrows and labels depict the 
derived performance scores for the various groups along the performance axis. 
In figures 5, 6, and 7 the experimental groups are plotted with the average 
derived performance score on the Y-axis. 

The presentation of the results of the detection of failures will be 
divided into three sections. The first presents the results for each mode of 
participation, and represents a replication of the Wickens and Kessel (Refer- 
ence 7) study with the between subjects design, the second examines the re- 
sults of the loading task, while the third reports the results of the trans- 
fer of training experiment. Group differences were analysed by means of a 
3-way Analysis of Variance -ANOVA (groups x dual task x experimental days). 



OF 


Figure 3: Effect of participatory mode and experimental condition 
on detection performance-single Task 
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Figure 5 

Detection performance as 


Figure 6 

• function of experimental condition 
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th.t intern.1 models developed sep.rot.ly tend to be -or. consletent, 1... 
verlable^and more een.ltiv. to s,.t« chonges. 

„ coopering the .ingle 

3 and 5) it is possible to determine whether ^ Figure8 3 and 

ter prior training in^the other m e of pa^ reduced somewhat, the over- 

5 «« can see that while this diff difference is also 

all MA superiority remains intact. « 
statistically reliable <F 1>10 “ 6.76, p < .05). 

Critic®l BM T2 1 2]£ 

the impact of the critical tracking tssk^y^be^evaluated^both^as^® 1 ^ 

fected detection performance ( F £8ure ) > ^ ? figure 7, it is 

performance of the priory 'w***** 1 ‘<255 a decrement in detection, 
evident that the introduction of the CT p somewhat more pro- 

I, might be expected, the decrement in the MA mod* there it a gub . 

nounced. While *******£ ^groups, equivalent to the decrement of both 
stantial decrement £ ° r J*®* U lI P a ^j the MA„-AU TT analyses, task load- 
the MA groups. For ^® h *J^I h ^I cf ^ ct (F „ « - #60, P < .05} F gQ 
ing showed a statistically f ll ^ l ***S b e not!9 however that the ptiWry 
5.45, P< .025 respectively). It should he the crltical taak , 

impact of this effect is lac *JJ* e J lf£lc lty i« V el, a point born out by fur- 
and not with the increase in it. “*“£ig a {™' 0 f ?he two dual task con- 

S3,y2rlS I ^ffiSSA collep.lng detection petiole over the 

two conditions in further analysis.) 

Figure . r.ve.1. th.t the crlticel tt*M •*[■"*" dii- 

trecklng perfoicnce, bo “V““J' “* £ .lone 1 indicted th.t 

Sct^tSS ”i»f 2 , 20 , < •«<»>• 

F Icily, Figure ® 

itr. «■ 

SdTt‘lTn‘cc«. in 1 *.ct did eerv. to divert .tt.ntlc.1 tenure, 
from the primary tracking/ detection process. 

{») Trensfer of Training 

. . ,l. mittive amount of transfer to the man - 

Manual Mode. In determining training, the MA TT group is compared 

ual mode resulting from jjrior autm *i essentially ttd no prior exper- 

wlth its control group MA, (Figure U wni 
fence in the failure detection task. 

W^.lly AB " tr.cklng'* perfoncc rede ncitected by critlcl 

task difficulty level. 
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Prom Figure* 3 through 7 it can be aeen that in general there it an 
overall MA superiority over MA for both tingle and dual task conditions. 
However the ANOVA failed to reveal these differences to be statistically re- 
liable. Examination of the data on a day by day basis reveals that the over- 
all HAj-MA__ difference is due to large differences that exist on day 1 which 
appear to dissipate couple tely when the two groups are compared on day 2 per- 
formance. This finding can be seen as support for the basic hypothesis that 
exposure to prior AU tracking and the development of an internal model based 
on visual cues only, produces only a small and transient facilitation of 
subsequent development of the internal model based on MA tracking. 

Automatic Mode. The degree of transfer resulting from prior MA train- 
ing to the AU mode is reflected in the performance of subjects in condition 
AU , and the comparison of this performance with that of the control group 
. Figure * 5 and 6, it is evident that the latter group 

faitda to Benefit at all from prior AU training, an observation supported by 
the lack of statistical reliability of the main effect when AU_._. and AU . . 
are compared. In marked contrast. Figures 5 and 6 suggest thai'nle AU n * C * 
group in fact showed considerable benefit from their prior MA training*£hen 
their performance is contrasted with that of the AU_ group. In Figure 5, 
the magnitude of this effect is seen to be considerably larger than the ef- 
fect for the control group or for the MA -MA__ contrast discussed in the pre- 
ceding section. i II r 


The statistical reliability of this improvement on the single task data 
wa * **«eased by a groups (AUj vs, AUj^)x days (Day 1 vs. Day 2) 2 x 2 ANOVA. 

Both main effects were statistically reliable. This indicates that (a) 
both groups Improved with practice (over two days) in their respective AU 
conditions <F. - 14.77, p < .001). (b) More crucially, from the viewpoint 

of the hypotheftts under investigation, the AU TT group performed reliably 

^i? e V hM ‘ did J he A ?I 8roup <P 1 10 " 5 * l9 » pI< • 05 >* lt 11 of course pos- 
•Ible to trgue that this effect resulted from greater exposure to and fami- 

Utrity with the overall experimental environment experienced by the AU 
group and not to transfer of the internal model. However this interprelltlon 
appears unlikely because the control group failed to show any such “genera l- 
ixed" transfer. 


w * can conclude that there is a transfer from MA to AU. The AU -AU 
differences are very large and statistically reliable and as such support 1 
the basic hypothesis that while there are different sets of cues operating 
the MA condition producea an internal model of the system that can be util- 
ised to advantage in subsequent automatic monitoring. 


SUMMARY AND CONCLUSIONS 


The major results can be summarised as follows: 

l) Detection of step increases in system order when the operator remaina 
in the control loop (MA node) is considerably faster and more accurate than 
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when he is removed (AU mode). This finding is consistent with both the find- 
ings of Young (Reference 2) and of Wickens end Kessel (Reference 7), 

2) The manual mode superiority was found to be more pronounced in this 
between subject design than the previous within subject study (Reference 7). 
This difference can be attributed to the fact that the subjects were allowed 
to develop separate internal models for either the manual or the automatic 
mode, thereby producing models that were always appropriate for the mode of 
participation employed. 

What is interesting in contrasting the two studies is the fact that AU 
performance is virtually Identical. The effect of the between-subjects mani- 
pulation Instead seems to have been to produce a large Improvement in MA 
detection. 

This result suggests that in the previous experiment the AU internal 
model was developed unhindered by the concurrent development of the MA inter- 
nal model while the reverse situation did not hold. It would appear that 
the development of the MA internal model in the previous experiment was some- 
how subject to interference from the AU model development, suggesting that 
subjects were paying attention to non-relevant, visual cues. It has been ar- 
gued (Reference 7) thet the sensitivity to proprioceptive information is 
reduced relative to visual information particularly when the two sources ere 
available at the same time and are conveying conflicting information (Refer- 
ences 13, 14, 15). In the AU mode the subjects have only visual cues as in- 
formation while in the MA mode both visual and proprioceptive information 
is available. Thus in the previous study, during the development of the MA 
Internal models there were times when these cues might be in conflict and 
subjects tended to fall back on the visual cues learned in the AU mode. This 
produced an over-emphasis on the visual cues and a subsequent degrading of 
the crucial proprioceptive Information. The Introduction of the between sub- 
ject design forced subjects to develop separate intemel models based upon 
the relevent cues available within each condition— a situation thet hes en- 
hanced the MA-AU differences found in the previous experiment, 

3) The overall MA superiority is evident In both single end dual task 
conditions. The effect of adding the Critical Task was to reduce the overall 
detection performance via a reduction In the accuracy of detections end an 
increase in response latencies. The impact of the second task was more marked 
for the MA condition than the AU condition. This result is consistent with 
the fact that the critical tracking task, placing heavy demands upon the sub- 
ject's response mechanism, produced an increase in interference at the struc- 
tural, motor level of performance in the MA mode that was not present in the 
AU mode of operation. Increasing the difficulty of the suberltleal loading 
task appeared to have little effect on detection performance in either mode, 
although it did serve to disrupt tracking performance. 

4) An analysis of the transfer of training experiment shows that there 
is very little transfer from the automatic mode to the manual mode. This fact 
adds further weight to the argument that the development of the internal model 
for the manual mode cannot utilize to advantage the internal model developed 
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£°r the automatic mode. The addition of the proprioceptive channele and eh* 

«E~ t J V % dMCtlbin8 £u " Cti0n in th * -»«•! -ode appear**to requi*e*the *de« 
velopment of a aaparate and unique Internal model. squire the de- 

the automatic* f °!JJf ***!.. t0 be P°* itlve tranafer from the manual mode to 
that JUit!**!'/ find f?L that oupporta the baalc hypotheala outlined above 
« tt iUt * nt ,etl o£ CUM operating, the MA modVprSucL 
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